Bucket testing, otherwise known as A/B testing, is in widespread use for testing web site product design. In one methodology, two different versions of a web site are run simultaneously to a selected bucket of users. Then, user traffic and activity are measured and the difference is noted to determine which design is preferred. One drawback to this method is that different users use different browsers. Assume we want to test a new feature specifically created for a bucket (a bucketed feature). Let's say we are testing the effectiveness of a new user interface (UI) interaction in a module in a specific bucket.
For example, we might want to test a new hover interaction in a module; i.e., when the user moves the mouse over that module something in the background changes. This feature might require a specific JavaScript library supported by most browsers with the exception of IE-6; in which case a random user who is assigned to this new bucket but using IE-6 will have a broken experience. To avoid that scenario, normally we introduce specific browser filters for each bucket so that only browsers which have capability to display that feature would fall into that bucket.
There is a need, however, for a system and method for better management of bucket testing.